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Abstract 

Background: Health professionals and policymakers aspire to make healthcare decisions based on the entire 
relevant research evidence. This, however, can rarely be achieved because a considerable amount of research 
findings are not published, especially in case of 'negative' results - a phenomenon widely recognized as publication 
bias. Different methods of detecting, quantifying and adjusting for publication bias in meta-analyses have been 
described in the literature, such as graphical approaches and formal statistical tests to detect publication bias, and 
statistical approaches to modify effect sizes to adjust a pooled estimate when the presence of publication bias is 
suspected. An up-to-date systematic review of the existing methods is lacking. 

Methods/design: The objectives of this systematic review are as follows: 

• To systematically review methodological articles which focus on non-publication of studies and to describe 
methods of detecting and/or quantifying and/or adjusting for publication bias in meta-analyses. 

• To appraise strengths and weaknesses of methods, the resources they require, and the conditions under which 
the method could be used, based on findings of included studies. 

We will systematically search Web of Science, Medline, and the Cochrane Library for methodological articles that 
describe at least one method of detecting and/or quantifying and/or adjusting for publication bias in meta-analyses. 
A dedicated data extraction form is developed and pilot-tested. Working in teams of two, we will independently 
extract relevant information from each eligible article. As this will be a qualitative systematic review, data reporting 
will involve a descriptive summary. 

Discussion: Results are expected to be publicly available in mid 2013. This systematic review together with the 
results of other systematic reviews of the OPEN project (To Overcome Failure to Publish Negative Findings) will serve 
as a basis for the development of future policies and guidelines regarding the assessment and handling of 
publication bias in meta-analyses. 
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Background 

Syntheses of published research, such as meta-analyses and 
systematic reviews, are becoming increasingly important in 
providing relevant and valid research evidence to clinical 
and health policy decision making. However, published 
studies might represent a biased selection of all studies that 
have been conducted, if statistically significant or positive' 
results are published preferably, a phenomenon widely 
known as publication bias [1-4]. When searching the litera- 
ture for meta-analyses, unpublished studies and studies 
published in the so called 'grey literature' only (such as con- 
ference abstracts, dissertations, policy documents, and book 
chapters) might be missed. The effect estimates of meta- 
analyses based exclusively on the published literature might 
be exaggerated and represent an overestimation of the true 
effect size [2,5], and consequently the patient might be ex- 
posed to an ineffective or even harmful treatment. 

Unfortunately, the elimination of publication bias can 
seldom be achieved, since relevant unpublished' studies 
are frequently difficult to find or not accessible. There are 
basically two kinds of unpublished' data. The first type of 
data, described as grey literature in the paragraph above, 
can still be identified through extended search strategies 
in computerized databases. The second type refers to data 
that have not been published at all and thus are far more 
difficult to identify. In order to tackle bias related to non- 
publication or distortion in the publication process of 
study findings there have been various calls for mandatory 
registration of clinical trials at inception [6]. In 2004, 
major medical journals agreed that they would only pub- 
lish trials that were previously registered [7]. However, 
some of the data fields requested in the registries are fre- 
quently incomplete [8]. Thus, until the complete registra- 
tion at inception of all trials is a well-established method 
and results of all trials are publicly available, it is of great 
importance to improve methods for detection, quantifica- 
tion and the adjustment for publication bias in meta- 
analyses and systematic reviews. 

In the literature various methods to detect, quantify and 
adjust for publication bias in meta-analyses have been de- 
scribed. There are graphical approaches, such as funnel 
plots [9], formal statistical tests to detect the presence of 
publication bias, such as the regression test proposed by 
Egger and colleagues [9], and statistical approaches to 
modify effect size to adjust pooled estimates when the 
presence of publication bias is suspected, such as the trim- 
and-fill method [10]. Still, statistical approaches to correct 
for missing studies are precarious. For instance, some au- 
thors criticize that the visual interpretation of a funnel 
plot depends too much on the subjective impression of 
the observer [11,12]. Furthermore, the performances of 
many of these methods have been evaluated using simula- 
tion studies, but concerns remain as to whether the simu- 
lations reflect real-life situations. 



Currently, consensus on what method is best to use only 
exists for the special case of tests for funnel plot asym- 
metry in meta-analyses of randomized controlled trials 
[13]. In order to inform the future development of policies 
and guidelines regarding the assessment of publication 
bias, we will conduct a systematic review of methods de- 
scribed in the literature. 



Objectives 

• To systematically review methodological articles 
which focus on non-publication of studies and to 
describe methods of detecting and/or quantifying 
and/or adjusting for publication bias in meta- 
analyses. 

• To appraise strengths and weaknesses of methods, 
the resources they require, and the conditions under 
which the method could be used, based on findings 
of included studies. 

This systematic review will be part of the OPEN project 
(To Overcome Failure to Publish Negative Findings) which, 
among other objectives, aims to elucidate the non-pub- 
lication of studies through a series of systematic reviews 
[14]. 

Methods 

Search strategy 

Literature searches for methodological studies are often 
difficult because of ill-defined boundaries and inappropri- 
ate indexing in commonly used bibliographic databases 
[15]. Previous experience by Song and colleagues [16] sug- 
gests that the most productive and efficient methods in- 
clude searching the Cochrane Methodology Register. In 
addition to the suggested Cochrane Methodology Register, 
we will conduct electronic literature search in Web of Sci- 
ence and Medline to identify relevant methodological arti- 
cles. Methodological articles are those that have developed 
or investigated methods for detecting, quantifying or 
adjusting for publication bias. As the MeSH term "publi- 
cation bias" has been introduced only in 1994, we decided 
to restrict our literature search to articles published be- 
tween 1994 and the present to facilitate our literature 
search. Key words used in the search of electronic data- 
bases will include: publication bias, file-drawer, and 
reporting bias. The search strategy has been developed 
with the support of a librarian/information specialist and 
we will focus on fully published articles. We will ask ex- 
perts in the field for any additional references and check 
references of included articles. No language restrictions 
will be applied. The full search strategies are displayed in 
Additional file 1. 
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Eligibility criteria 

Methodological articles will be considered eligible for in- 
clusion if they describe a method for at least one of the 
following tasks: i) detection, ii) quantification, or iii) ad- 
justment for publication bias in meta-analyses. 

We will exclude original clinical trial reports, observa- 
tional clinical studies, and clinical systematic reviews. 

Study selection 

Two reviewers will independently and in duplicate 
screen titles and abstracts of search results. If both re- 
viewers do not agree on exclusion based on its title and 
abstract, the full text will be retrieved and assessed for 
eligibility. Any disagreement among reviewers will be re- 
solved by discussion and consensus or, if needed, arbitra- 
tion by a third reviewer. 

Data extraction 

A dedicated data extraction form will be used (Additional 
file 2), and two reviewers will independently extract the 
following information: 

O Basic data: 
O author names 
O language 
O year of publication 
O journal name 

O type of report (for example, narrative review, 
systematic review, methodological study and so on) 

O study objectives 

O funding source 
O On methods to detect and/or quantify and/or 

adjust for publication bias in meta-analyses: 

O Short description of the method 

O What form of bias the method pays attention to 

O Underlying assumptions 

O Purpose of the method 

O Resources required for using the method 

O Strengths and weaknesses of the methods 
described (as discussed in the article) 

O If the method has been applied to meta-analyses 
with real world datasets, or to a dataset for which 
one can be reasonably confident that all studies 
conducted have been included (for example, 
datasets from trial registries from medical 
regulatory authorities such as the Food and Drug 
Administration (FDA) yes the definition is correct 
or the European Medicines Agency (EMA)). 

The full data extraction sheet is displayed in Additional 
file 2. 

Two reviewers will extract relevant data from each of 
the included methodological articles independently and 
in duplicate. Any disagreements will be resolved by 



discussion and consensus or, if needed, arbitration by a 
third reviewer. 

Data analysis and reporting 

A short definition of the method described to detect and/ 
or quantify and/or adjust for publication bias in meta- 
analyses will be given. Each method will be classified as 
described by the author. If the author does not propose a 
classification, we will categorize the method based on a 
standardized method classification sheet. 
The different categories comprise: 

• Study registration 

• Literature search 

• Funnel plot 

O Tests for funnel plot asymmetry 
O Methods to adjust for publication bias based on 
funnel plots (trim and fill) 

• Selection models 

• Selection models with data augmentation 

• Sensitivity analyses based on selection models 

• New statistical approaches 

• Updating reviews 

• Publication process 

• Research ethics/policy 

• Confirmatory studies 

• Other: 



Available methods will be critically appraised in terms 
of underlying assumptions, conditions under which the 
method could be used, usefulness, limitations, and re- 
sources required. This appraisal will be based on the de- 
scription provided in the included studies. 

To assess the validity of the method, we will describe if 
the method has been tested in an empirical dataset for 
which one can be reasonably confident that all studies 
conducted have been included (for example, datasets from 
trial registries from medical regulatory authorities such as 
the FDA or EMA). 

Data extracted from the included studies and results of 
critical appraisal will be presented in tables and described 
narratively. We will report this systematic review accor- 
ding to PRISMA (Preferred Reporting Items for System- 
atic Reviews and Meta- Analyses) guidelines [17]. 

Discussion 

We aim to provide a comprehensive systematic review of 
the various methods to detect and/or quantify and/or ad- 
just for publication bias, as they are used and described in 
the literature. Furthermore, we will describe the various 
methods found and illustrate the strengths and weak- 
nesses of each method in this systematic review. 

Our protocol has strengths and limitations. A strength 
of our protocol is the systematic approach to identify 
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methodological articles through a sensitive search strategy, 
which includes databases previously tested for the search 
of methodological studies on publication bias [16]. A limi- 
tation of this protocol is the assessment of the various 
methods based only on the information provided in the 
methodological articles themselves without applying them 
to a real world clinical dataset 

As part of the OPEN project, this systematic review 
aims to raise awareness of the importance of bias related 
to non-publication or distortion in the publication process 
of research findings and the complexity of this issue. This, 
and other systematic reviews conducted in the OPEN pro- 
ject, will also provide a foundation for a recommendations 
workshop, during which key members of the biomedical 
research community (for example, flinders, research ethics 
committees, journal editors) will develop future policies 
and guidelines to tackle the non-publication of biomedical 
research findings and related biases. 

Additional files 



Additional file 1: Search results and search strategies. 
Additional file 2: Data extraction sheet. 
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